Unsupervised integration test builder

ABSTRACT

A system and method of generating one or more integration tests are disclosed herein. A computing system receives a URL from a client device. The URL corresponds to a website hosted by a third party server. The computing system generates a recurrent neural network model for testing of the website. The one or more variables associated with the recurrent neural network model are defined by a genetic algorithm. The computing system inputs code associated with the website into the recurrent neural network model. The recurrent neural network model learns a plurality of possible paths through the website by permutating through each possible set of options on the website. The recurrent neural network mode generates, as output, a plurality of integration tests for at least the test website. The computing system compiles the plurality of integration tests into a format compatible with a testing service specified by the client device.

CROSS-REFERENCE TO RELATED APPLICATION INFORMATION

This is a continuation of U.S. patent application Ser. No. 16/697,561,filed Nov. 27, 2019, which is incorporated herein by reference in itsentirety.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to a system and method ofgenerating integration tests for a website.

BACKGROUND

Testing a website through a web interface is often difficult, eitherrequiring testers to manually test the site, developers buildingintegration tests, or merely skipping the tests altogether. Suchoperations are typically costly, as it takes either a long time to buildunit tests, or companies lose money for having broken site links, forms,and the like.

SUMMARY

In some embodiments, a method of generating one or more integrationtests is disclosed herein. A computing system receives a uniformresource locator (URL) from a client device. The URL corresponds to awebsite hosted by a third party server. The computing system generates arecurrent neural network model for testing of the website. The one ormore variables associated with the recurrent neural network model aredefined by a genetic algorithm. The computing system inputs codeassociated with the website into the recurrent neural network model. Therecurrent neural network model learns a plurality of possible pathsthrough the website by permutating through each possible set of optionson the website. The recurrent neural network mode generates, as output,a plurality of integration tests for at least the test website. Thecomputing system compiles the plurality of integration tests into aformat compatible with a testing service specified by the client device.

In some embodiments, the computing system uploads the plurality ofintegration tests in the compatible format into the testing servicespecified by the client device.

In some embodiments, the computing system receives, from the clientdevice, a second URL corresponding to a web page. The computing systeminputs code associated with the second URL into the recurrent neuralnetwork model. The recurrent neural network identifies a secondplurality of possible permutations through the web page, based on thelearned plurality of possible permutations.

In some embodiments, the computing system receives, from the clientdevice, a second URL corresponding to an update of a web page of thewebsite. The computing system inputs code associated with the second URLinto the recurrent neural network model. The recurrent neural networkidentifies a plurality of additional possible permutations through theweb page. The plurality of additional possible permutations are distinctfrom the plurality of permutations prior to the update.

In some embodiments, the computing system executes the one or moreintegration tests on the update of the web page.

In some embodiments, learning, by the recurrent neural network model,the plurality of possible permutations through the website bypermutating through each possible set of options on the website includeslearning one or more possible paths through the website to not take forfuture testing.

In some embodiments, the computing system receives, from the clientdevice, a second URL corresponding to a web page. The computing systeminputs code associated with the second URL into the recurrent neuralnetwork model. The recurrent neural network identifies a secondplurality of possible permutations through the web page, based on thelearned plurality of possible permutations by skipping one or more setsof options of the webpage that lead to dead ends.

In some embodiments, the one or more variables include a speed ofswitching between states on the website.

In some embodiments, a system is disclosed herein. The system includes aprocessor and a memory. The memory has programming instructions storedthereon, which, when executed by the processor, perform one or moreoperations. The one or more operations include generating a plurality ofintegration tests. Generating a plurality of integration tests comprisesreceiving a uniform resource locator (URL) from a client device, the URLcorresponding to a test website, generating a recurrent neural networkmodel for testing of the test website, wherein one or more variablesassociated with the recurrent neural network model are defined by agenetic algorithm, inputting associated with the test website into therecurrent neural network model, learning, by the recurrent neuralnetwork model, a plurality of possible paths through the test website bypermutating through each possible set of options on the website, andoutputting, by the recurrent neural network model, the plurality ofintegration tests. The one or more operations further include testing acandidate website using the plurality of integration tests. Testing thecandidate website using the plurality of integration tests comprisescompiling the plurality of integration tests into a format compatiblewith a testing service and executing the plurality of integration testsin the compatible format on code associated with the candidate website.

In some embodiments, the one or more operations further includereceiving, from the client device, a second URL corresponding to a webpage, inputting code associated with the second URL into the recurrentneural network model, and identifying, by the recurrent neural network,a second plurality of possible permutations through the web page, basedon the learned plurality of possible permutations.

In some embodiments, the one or more operations further includereceiving, from the client device, a second URL corresponding to anupdate of a web page of the website, inputting code associated with thesecond URL into the recurrent neural network model, and identifying aplurality of additional possible permutations through the web page. Theplurality of additional possible permutations are distinct from theplurality of permutations prior to the update.

In some embodiments, the one or more operations further includeexecuting the one or more integration tests on the update of the webpage.

In some embodiments, learning, by the recurrent neural network model,the plurality of possible permutations through the website bypermutating through each possible set of options on the website includeslearning one or more possible paths through the website to not take forfuture testing.

In some embodiments, the one or more operations further includereceiving, by the computing system from the client device, a second URLcorresponding to a web page, inputting, by the computing system, codeassociated with the second URL into the recurrent neural network model,and identifying, by the recurrent neural network, a second plurality ofpossible permutations through the web page, based on the learnedplurality of possible permutations by skipping one or more sets ofoptions of the webpage that lead to dead ends.

In some embodiments, the one or more variables include a speed ofswitching between states on the website.

In some embodiments, a non-transitory computer-readable medium isdisclosed herein. The non-transitory computer-readable medium includesone or more sequences of instructions that, when executed by one or moreprocessors, causes one or more operations. A computing system receives auniform resource locator (URL) from a client device. The URL correspondsto a website hosted by a third party server. The computing systemgenerates a recurrent neural network model for testing of the website.The one or more variables associated with the recurrent neural networkmodel are defined by a genetic algorithm. The computing system inputscode associated with the website into the recurrent neural networkmodel. The recurrent neural network model learns a plurality of possiblepaths through the website by permutating through each possible set ofoptions on the website. The recurrent neural network model generates, asoutput, a plurality of integration tests for at least the test website.The computing system compiles the plurality of integration tests into aformat compatible with a testing service specified by the client device.

In some embodiments, the computing system uploads the plurality ofintegration tests in the compatible format into the testing servicespecified by the client device.

In some embodiments, the computing system receives, from the clientdevice, a second URL corresponding to a web page. The computing systeminputs code associated with the second URL into the recurrent neuralnetwork model. The recurrent neural network identifies a secondplurality of possible permutations through the web page, based on thelearned plurality of possible permutations.

In some embodiments, the computing system receives, from the clientdevice, a second URL corresponding to an update of a web page of thewebsite. The computing system inputs code associated with the second URLinto the recurrent neural network model. The recurrent neural networkidentifies a plurality of additional possible permutations through theweb page. The plurality of additional possible permutations are distinctfrom the plurality of permutations prior to the update.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentdisclosure can be understood in detail, a more particular description ofthe disclosure, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrated onlytypical embodiments of this disclosure and are therefore not to beconsidered limiting of its scope, for the disclosure may admit to otherequally effective embodiments.

FIG. 1 is a block diagram illustrating a computing environment,according to example embodiments.

FIG. 2 is a flow diagram illustrating a method of generating anintegration test, according to example embodiments.

FIG. 3 is a flow diagram illustrating a method of identifying possiblepaths through a website, according to example embodiments.

FIG. 4 is a block diagram visually representing various paths throughwebsite, according to example embodiments.

FIG. 5 is a block diagram illustrating a computing environment,according to example embodiments.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures. It is contemplated that elements disclosed in oneembodiment may be beneficially utilized on other embodiments withoutspecific recitation.

DETAILED DESCRIPTION

One or more techniques disclosed herein generally relate to a system andmethod of generating integration tests for a website. Buildingintegration tests for a website is conventionally time consuming,costly, and always changing. One or more techniques described hereineliminates these issues by using a combination of genetic algorithms,recurrent neural networks (RNNs), and prior knowledge of state (e.g.,prior genetic version of the website) to generate integration testsautomatically, with little or no user interaction. For example, one ormore techniques disclosed herein involve implementing an algorithm thatrandomly selects web elements and attempts to interact with those webelements (e.g., testing user log-in attempts). Over time, the algorithmmay learn various “paths” through the website, generating tests, andidentifying broken links, dead ends, and the like.

The term “user” as used herein includes, for example, a person or entitythat owns a computing device or wireless device; a person or entity thatoperates or utilizes a computing device; or a person or entity that isotherwise associated with a computing device or wireless device. It iscontemplated that the term “user” is not intended to be limiting and mayinclude various examples beyond those described.

FIG. 1 is a block diagram illustrating a computing environment 100,according to one embodiment. Computing environment 100 may include atleast a client device 102, an organization computing system 104, one ormore third party web servers 106 (hereinafter “third party web server106”), a testing service 110, and a database 110 communicating vianetwork 105.

Network 105 may be of any suitable type, including individualconnections via the Internet, such as cellular or Wi-Fi networks. Insome embodiments, network 105 may connect terminals, services, andmobile devices using direct connections, such as radio frequencyidentification (RFID), near-field communication (NFC), Bluetooth™,low-energy Bluetooth™ (BLE), Wi-Fi™, ZigBee™, ambient backscattercommunication (ABC) protocols, USB, WAN, or LAN. Because the informationtransmitted may be personal or confidential, security concerns maydictate one or more of these types of connection be encrypted orotherwise secured. In some embodiments, however, the information beingtransmitted may be less personal, and therefore, the network connectionsmay be selected for convenience over security.

Network 105 may include any type of computer networking arrangement usedto exchange data or information. For example, network 105 may be theInternet, a private data network, virtual private network using a publicnetwork and/or other suitable connection(s) that enables components incomputing environment 100 to send and receive information between thecomponents of system 100.

Client device 102 may be operated by a user. For example, client device102 may be a mobile device, a tablet, a desktop computer, or anycomputing system having the capabilities described herein. Client device102 may belong to or be provided to a user or may be borrowed, rented,or shared. Users may include, but are not limited to, individuals suchas, for example, subscribers, clients, prospective clients, or customersof an entity associated with organization computing system 104, such asindividuals who have obtained, will obtain, or may obtain a product,service, or consultation from an entity associated with organizationcomputing system 104. Generally, client device 102 may be associatedwith third party web server 106.

Client device 102 may include at least application 112. Application 112may be representative of a web browser that allows access to a websiteor a stand-alone application. Client device 102 may access application112 to access functionality of organization computing system 104. Forexample, client device 102 may access application 112 to access anintegration test generator 116 hosted by organization computing system104.

Further, client device 102 may communicate over network 105 to request awebpage, for example, from third party web server 106. For example,client device 102 may be configured to execute application 112 to accesscontent managed by web third party web server 106. The content that isdisplayed to client device 102 may be transmitted from third party webserver 106 to client device 102, and subsequently processed byapplication 112 for display through a graphical user interface of clientdevice 102.

Organization computing system 104 may include at least web clientapplication server 114 and integration test generator 116. Integrationtest generator 116 may be comprised of one or more software modules. Theone or more software modules may be collections of code or instructionsstored on a media (e.g., memory of organization computing system 104)that represent a series of machine instructions (e.g., program code)that implements one or more algorithmic steps. Such machine instructionsmay be the actual computer code the processor of organization computingsystem 104 interprets to implement the instructions or, alternatively,may be a higher level of coding of the instructions that is interpretedto obtain the actual computer code. The one or more software modules mayalso include one or more hardware components. One or more aspects of anexample algorithm may be performed by the hardware components (e.g.,circuitry) itself, rather as a result of an instructions.

Integration test generator 116 may be configured to analyze a web siteuploaded by a user (e.g., client device 102), and generate anintegration test based on the analysis. For example, integration testgenerator 116 may be configured to generate an integration test withlittle or no input from the user. Integration test generator 116 mayinclude one or more recurrent neural network (RNN) models 120(hereinafter “RNN model 120”) and one or more genetic algorithms 122(hereinafter “genetic algorithms 122”).

RNN model 120 may be configured to maintain a current and previousstates of a given website. For example, RNN model 120 may be trained byintegration test generator 116. Integration test generator 116 may trainRNN model 120 by providing RNN model 120 with a plurality of scripts fora plurality of web sites. RNN model 120 may analyze the scripts for theplurality of websites to identify one or more paths contained in thewebsite. For example, given a certain web page of a website, RNN model120 may learn the various ways a user may interact with the web page,the various locations a user can navigate from the web page, variousdead ends of the web page, and the like. Further, RNN model 120 maylearn how to determine which paths were previously taken through thewebsite, such that RNN model 120 does not repeat paths. Accordingly,once trained, RNN model 120 may be able to iterate through all possiblepaths of a website, while being cognizant of the current state of thewebsite and previous states of the website.

Genetic algorithms 122 may be configured to inject randomness into theanalysis performed by RNN model 120. In other words, genetic algorithms122 may provide hyperparameter tuning of RNN model 120. For example, insome embodiments, genetic algorithms 122 may define the speed at whichRNN model 120 simulates a user clicking through a web site. In someembodiments, genetic algorithms 122 may define a number of layers in RNNmodel 120. In some embodiments, genetic algorithms 122 may define aspeed at which RNN model 120 switches between states (e.g., one second).In some embodiments, genetic algorithms 122 may define the number offree variables in RNN model 120. In some embodiments, genetic algorithms122 may define a number of long short-term memory (LSTM) modules in RNNmodel 120. in some embodiments, genetic algorithms 122 may define whattype of threads are analyzed (e.g., cascading style sheets (CSS),hypertext markup language (HTML), field elements, a combination thereof,and the like). In some embodiments, genetic algorithms 122 may define adepth of the search (e.g., how many states RNN model 120 may iteratethrough). Generally, genetic algorithms 122 may be defined by an enduser via application 112 of client device 102.

In addition to identifying current paths through a website, integrationtest generator 116 may further be configured to analyze beta pages orupdates to a given website. For example, if a user wants to test a betapage or update to a web page prior to rolling out the web page,integration test generator 116 may be able to iterate through the betapage or update to identify any new paths through the website or anybroken paths (e.g., dead end) that may be caused by the beta page orupdate.

As output, integration test generator 116 may generate a plurality ofintegration tests for the provided website. Integration test generator116 may compile the plurality of integration tests into a formatcompatible with a testing service provided by client device 102.Exemplary testing services may include, for example, Selenium, Katalon,Unified Functional Testing (UFT), and the like. In some embodiments,integration test generator 116 may interface directly with a testingservice 108 by transmitting the compiled integration tests directlythereto. In some embodiments, integration test generator 116 may providethe compiled integration tests directly to client device 102, thusallowing the end user to interact with testing service 108.

Third party web server 106 may be representative of one or morecomputing devices configured to host one or more websites. For purposesof this discuss, assume that third party web server 106 is a web serverassociated with client device 102. Third party web server 106 mayinclude one or more web pages 130 of a web site and one or more updates132 (or beta pages) of a website. When a user of client device 102, forexample, wishes to test a website, user may submit a URL associated witha website hosted by third party web server 106 to integration testgenerator 116 for analysis. In some embodiments, the URL may beassociated with a web page 130 of the one or more web pages. Each update132 may correspond to a web page that has not been rolled out, or madeavailable to, users on the open Internet. In some embodiments, a user ofclient device 102 may test updates 132 by providing a URL (e.g., aprivate URL) to integration test generator 116, such that integrationtest generator 116 may iterate through the various paths resulting fromupdate 132.

Referring back to organization computing system 104, organizationcomputing system 104 may be associated with database 110. Database 110may include one or more user profile 124 (hereinafter “user profile124”). User profile 124 may be associated with a specific end user or anorganization that encompasses a plurality of end users. User profile 124may include one or more paths 126, one or more results 128, and one ormore integration tests 131.

Each path 126 may be representative of one possible path through awebsite hosted by a third part web server 106. Database 110 may maintainpaths 126 for a given website, for example, so that integration testgenerator 116 may determine if an update 132 or beta page breaks apreviously determined path. Results 128 may correspond to variousresults determined by RNN model 120. For example, RNN model 120 maydetermine that a given update 132 or beta page breaks a previouslydetermined path, that there is a dead end via an existing website path,and the like. For example, RNN model 120 may identify a series ofactions, such as selecting form fields and filling in username andpasswords, to see if a user can login. In another example, RNN model 120may identify a set of actions corresponding to a user trying to make anonline payment. RNN model 120 may intentionally miss fields in theonline payment process to see if the system catches such errors.Essentially, RNN model 120 may learn a series of CSS/id selectors andattempt to utilize them in the analysis. Integration tests 131 may berepresentative of one or more compiled integration tests generated byintegration test generator 116.

FIG. 2 is a flow diagram illustrating a method 200 of generating anintegration test, according to example embodiments. Method 200 may beginat step 202.

At step 202, organization computing system 104 may receive a uniformresource locator (URL) from client device 102. For example, clientdevice 102 may submit URL for a web page 130 or website hosted by thirdparty web server 106 to organization computing system 104 viaapplication 112 executing thereon.

At step 204, organization computing system 104 may generate an RNN model120 for testing of the website. In some embodiments, RNN model 120 maybe trained prior to receipt of the URL. For example, integration testgenerator 116 may train RNN model 120 using various websites prior toallowing users access to functionalities thereof. In some embodiments,integration test generator 116 may continually train RNN model 120 asnew requests are received. Generally, RNN model 120 may be trained toidentify all possible paths through a given website. For example, giveninput parameters, RNN model 120 may be configured to iterate throughevery possible option for a given website.

At step 206, organization computing system 104 may input code (e.g., webscripts) associated with the website into RNN model 120. At step 208,RNN model 120 may learn a plurality of possible paths through thewebsite by permutating through each possible set of options on thewebsite. In some embodiments, the way in which RNN model 120 learns theplurality of possible paths is hyper-parameterized by one or moregenetic algorithms. For example, integration test generator 116 may tuneRNN model 120 via one or more genetic algorithms 122. Genetic algorithms122 may define the way in which RNN model 120 iterates through thewebsite. In some embodiments, such parameters may be defined by an enduser when client device 102 uploads URL to integration test generator116.

At step 210, organization computing system 104 may generate a pluralityof integration test for the website. For example, as a result of RNNmodel 120 iterating through possible paths of the website, integrationtest generator 116 may generate one or more integration test for thewebsite.

At step 212, organization computing system 104 may compile the pluralityof integration tests into a format compatible with a resting servicespecified by client device 102. For example, integration test generator116 may compile the plurality of integration tests into a formatcompatible with one of Selenium, Katalon, Unified Functional Testing(UFT), and the like.

FIG. 3 is a flow diagram illustrating a method 300 of identifyingpossible paths through a website, according to example embodiments.Method 300 may begin at step 302.

At step 302, organization computing system 104 may receive a URL fromclient device 102. For example, client device 102 may submit URL for aweb page 130 or website hosted by third party web server 106 toorganization computing system 104 via application 112 executing thereon.

At step 304, organization computing system 104 may input code associatedwith the website into RNN model 120. At step 306, RNN model 120 maylearn a plurality of possible paths through the website by permutatingthrough each possible set of options on the website. In someembodiments, the way in which RNN model 120 learns the plurality ofpossible paths is hyper-parameterized by one or more genetic algorithms.For example, integration test generator 116 may tune RNN model 120 viaone or more genetic algorithms 122. Genetic algorithms 122 may definethe way in which RNN model 120 iterates through the website. In someembodiments, such parameters may be defined by an end user when clientdevice 102 uploads URL to integration test generator 116.

At step 308, organization computing system 104 may receive a second URLfrom client device 102. For example, client device 102 may submit URLfor an update 132, or beta page, for website hosted by third party webserver 106 to organization computing system 104 via application 112executing thereon.

At step 310, organization computing system 104 may input code (e.g., webscripts) associated with update 132 into RNN model 120. At step 312, RNNmodel 120 may learn a plurality of possible new paths through thewebsite by permutating through each possible set of options on thewebsite with update 132 incorporated therein. RNN model 120 may comparedetermined paths to paths 126 stored in database 110 to determine thosepaths that are new. In some embodiments, the way in which RNN model 120learns the plurality of possible paths is hyper-parameterized by one ormore genetic algorithms. For example, integration test generator 116 maytune RNN model 120 via one or more genetic algorithms 122. Geneticalgorithms 122 may define the way in which RNN model 120 iteratesthrough the website. In some embodiments, such parameters may be definedby an end user when client device 102 uploads URL to integration testgenerator 116. RNN model 120 may compare determined paths to paths 126stored in database 110 to determine those paths that are new.

FIG. 4 is a block diagram 400 visually representing various pathsthrough website, according to example embodiments. The example providedin block diagram 400 is of a user interacting with a log-in prompt.

As shown, initially, at block 402, the user is presented with a log-inprompt. The log-in prompt includes a username field, a password field,and a login actionable element (e.g., submit button). RNN model 120 mayiterate through every possible path given the initial state illustratedin block 402.

From block 402, RNN model 120 may take three paths, illustrated by block404, block 414, and block 424. In block 404, RNN model 120 firstinteracts with the username field. For example, RNN model 120 may fillin the username field with a test username. From block 404, RNN model120 may follow two paths: block 406 and block 412. In block 406, RNNmodel 120 may interact with the password field. For example, RNN model120 may fill in the password field with a test password. The only pathfollowing block 406 is to proceed to block 408. At block 408, RNN model120 may interact with the login actionable element. Block 410 mayrepresent a scrip of a successful path. Referring back to block 404, RNNmodel 120 may subsequently interact with block 412. In block 412, RNNmodel 120 may interact with the login actionable element, instead offirst filling in the password field. As those skilled in the artrecognize, such a submission (e.g., a username without a password) wouldresult in an error. Accordingly, such path is a “dead end.”

Referring back to block 402, RNN model 120 may proceed to block 414. Inblock 414, RNN model 120 may first interact with the password field. Forexample, RNN model 120 may fill in the password field with a testpassword. From block 414, RNN model 120 may follow two possible paths:block 416 and block 422. In block 416, RNN model 120 may interact withthe username field. For example, RNN model 120 may fill in the usernamefield with a test username. The only path following block 416 is toproceed to block 418. At block 408, RNN model 120 may interact with thelogin actionable element. Block 420 may represent a scrip of asuccessful path. Referring back to block 414, RNN model 120 maysubsequently interact with block 422. In block 422, RNN model 120 mayinteract with the login actionable element, instead of first filling inthe username field. As those skilled in the art recognize, such asubmission (e.g., a password without a username) would result in anerror. Accordingly, such path is a “dead end.”

Referring back to block 402, RNN model 120 may proceed to block 424. Inblock 424, RNN model 120 may interact with login actionable element,instead of first filling in the username or password fields. As thoseskilled in the art recognize, such a submission (e.g., no username andno password) would result in an error. Accordingly, such path is a “deadend.”

FIG. 5 is a block diagram illustrating an exemplary computingenvironment 500, according to some embodiments. Computing environment500 includes computing system 502 and computing system 552. Computingsystem 502 may be representative of client device 102. Computing system552 may be representative of organization computing system 104.

Computing system 502 may include a processor 504, a memory 506, astorage 508, and a network interface 510. In some embodiments, computingsystem 502 may be coupled to one or more I/O device(s) 512 (e.g.,keyboard, mouse, etc.).

Processor 504 may retrieve and execute program code 520 (i.e.,programming instructions) stored in memory 506, as well as stores andretrieves application data. Processor 504 may be included to berepresentative of a single processor, multiple processors, a singleprocessor having multiple processing cores, and the like. Networkinterface 510 may be any type of network communications allowingcomputing system 502 to communicate externally via computing network505. For example, network interface 510 is configured to enable externalcommunication with computing system 552.

Storage 508 may be, for example, a disk storage device. Although shownas a single unit, storage 508 may be a combination of fixed and/orremovable storage devices, such as fixed disk drives, removable memorycards, optical storage, network attached storage (NAS), storage areanetwork (SAN), and the like.

Memory 506 may include application 516, operating system 518, andprogram code 520. Program code 520 may be accessed by processor 504 forprocessing (i.e., executing program instructions). Program code 520 mayinclude, for example, executable instructions for communicating withcomputing system 552 to display one or more pages of website 564.Application 516 may enable a user of computing system 502 to access afunctionality of computing system 552. For example, application 516 mayaccess content managed by computing system 552, such as integration testgenerator 570. The content that is displayed to a user of computingsystem 502 may be transmitted from computing system 552 to computingsystem 502, and subsequently processed by application 516 for displaythrough a graphical user interface (GUI) of computing system 502.

Computing system 552 may include a processor 554, a memory 556, astorage 558, and a network interface 560. In some embodiments, computingsystem 552 may be coupled to one or more I/O device(s) 562. In someembodiments, computing system 552 may be in communication with database110.

Processor 554 may retrieve and execute program code 566 (i.e.,programming instructions) stored in memory 556, as well as stores andretrieves application data. Processor 554 is included to berepresentative of a single processor, multiple processors, a singleprocessor having multiple processing cores, and the like. Networkinterface 560 may be any type of network communications enablingcomputing system 552 to communicate externally via computing network505. For example, network interface 560 allows computing system 552 tocommunicate with computer system 502.

Storage 558 may be, for example, a disk storage device. Although shownas a single unit, storage 558 may be a combination of fixed and/orremovable storage devices, such as fixed disk drives, removable memorycards, optical storage, network attached storage (NAS), storage areanetwork (SAN), and the like.

Memory 556 may include website 564, operating system 566, program code568, integration test generator 570, and testing service 572. Programcode 568 may be accessed by processor 554 for processing (i.e.,executing program instructions). Program code 568 may include, forexample, executable instructions configured to perform steps discussedabove in conjunction with FIGS. 2-4. As an example, processor 554 mayaccess program code 568 to perform operations related to generatingintegration test or identifying new paths through a website given anupdate or beta page. Website 564 may be accessed by computing system502. For example, website 564 may include content accessed by computingsystem 502 via a web browser or application.

Integration test generator 570 may be configured to analyze a web siteuploaded by a user (e.g., computing system 502), and generate anintegration test based on the analysis. For example, integration testgenerator 570 may be configured to generate an integration test withlittle or no input from the user. Integration test generator 570 mayinclude one or more RNN models and one or more genetic algorithms. RNNmodel may be configured to maintain a current and previous states of agiven website. For example, RNN model may be trained by integration testgenerator 570. Integration test generator 570 may train RNN model byproviding RNN model with a plurality of scripts for a plurality of websites. RNN model may analyze the scripts for the plurality of websitesto identify one or more paths contained in the website. For example,given a certain web page of a website, RNN model may learn the variousways a user may interact with the web page, the various locations a usercan navigate from the web page, various dead ends of the web page, andthe like. Further, RNN model may learn how to determine which paths werepreviously taken through the website, such that RNN model does notrepeat paths. Accordingly, once trained, RNN model may be able toiterate through all possible paths of a website, while being cognizantof the current state of the website and previous states of the website.Genetic algorithms may be configured to inject randomness into theanalysis performed by RNN model. In other words, genetic algorithms mayprovide hyperparameter tuning of RNN model.

Testing service 572 may receive a compiled integration tests fromintegration test generator 570. Testing service 572 may execute thecompiled integration test to test and analyze the website. Althoughtesting service 572 is shown as part of computing system 552, thoseskilled in the art recognize, that testing service 572 may beindependent from computing system 552.

While the foregoing is directed to embodiments described herein, otherand further embodiments may be devised without departing from the basicscope thereof. For example, aspects of the present disclosure may beimplemented in hardware or software or a combination of hardware andsoftware. One embodiment described herein may be implemented as aprogram product for use with a computer system. The program(s) of theprogram product define functions of the embodiments (including themethods described herein) and can be contained on a variety ofcomputer-readable storage media. Illustrative computer-readable storagemedia include, but are not limited to: (i) non-writable storage media(e.g., read-only memory (ROM) devices within a computer, such as CD-ROMdisks readably by a CD-ROM drive, flash memory, ROM chips, or any typeof solid-state non-volatile memory) on which information is permanentlystored; and (ii) writable storage media (e.g., floppy disks within adiskette drive or hard-disk drive or any type of solid staterandom-access memory) on which alterable information is stored. Suchcomputer-readable storage media, when carrying computer-readableinstructions that direct the functions of the disclosed embodiments, areembodiments of the present disclosure.

It will be appreciated to those skilled in the art that the precedingexamples are exemplary and not limiting. It is intended that allpermutations, enhancements, equivalents, and improvements thereto areapparent to those skilled in the art upon a reading of the specificationand a study of the drawings are included within the true spirit andscope of the present disclosure. It is therefore intended that thefollowing appended claims include all such modifications, permutations,and equivalents as fall within the true spirit and scope of theseteachings.

What is claimed:
 1. A method of testing a target website, comprising:identifying, by a computing system, the target web site hosted by athird party server; inputting, by the computing system, code associatedwith the target web site into a recurrent neural network, wherein agenetic algorithm tunes hyperparameters of the recurrent neural networkthat define how the recurrent neural network simulates a user navigatingthe target website; learning, by the computing system, a first pluralityof paths through the target web site in accordance with the geneticalgorithm; and based on the learning, identifying, by the computingsystem, a subset of broken paths that result in an error.
 2. The methodof claim 1, further comprising: identifying, by the computing system, auniform resource locator (URL) corresponding to the target website;inputting, by the computing system, further code associated with the URLinto the recurrent neural network; and identifying, by the recurrentneural network, a second plurality of paths through the target web site,based on the learned first plurality of paths.
 3. The method of claim 1,further comprising: determining, by the computing system, that thetarget website underwent an update; inputting, by the computing system,further code associated with the updated target website into therecurrent neural network; and identifying, by the recurrent neuralnetwork, a second plurality of paths through the target web site, thesecond plurality of paths distinct from the first plurality of pathsprior to the update.
 4. The method of claim 1, further comprising:receiving, by the computing system, a uniform resource locator (URL)corresponding to the target website; inputting, by the computing system,further code associated with the URL into the recurrent neural network;and identifying, by the recurrent neural network, a second plurality ofpaths through the target website, based on the learned first pluralityof paths by skipping one or more sets of options of the target websitethat correspond to the learned subset of broken paths that result in anerror.
 5. The method of claim 1, wherein the hyperparameters include aspeed of switching between states on the target website.
 6. The methodof claim 1, further comprising: generating, by the computing system, aplurality of integration tests for at least the target website based onthe learning.
 7. The method of claim 1, further comprising: training, bythe computing system, the recurrent neural network to learn pathsthrough websites using training data sets comprising code associatedwith the websites.
 8. A non-transitory computer readable mediumincluding one or more sequences of instructions that, when executed byone or more processors, causes a computing system to perform operations,comprising: receiving, by the computing system from a client device, atarget website hosted by a third party server; inputting, by thecomputing system, code associated with the target web site into arecurrent neural network, wherein a genetic algorithm tuneshyperparameters of the recurrent neural network that define how therecurrent neural network simulates a user navigating the target website;learning, by the computing system, a first plurality of paths throughthe target web site in accordance with the genetic algorithm; and basedon the learning, identifying, by the computing system, a subset ofbroken paths that result in an error.
 9. The non-transitory computerreadable medium of claim 8, further comprising: receiving, by thecomputing system from the client device, a uniform resource locator(URL) corresponding to the target website; inputting, by the computingsystem, further code associated with the URL into the recurrent neuralnetwork; and identifying, by the recurrent neural network, a secondplurality of paths through the target web site, based on the learnedfirst plurality of paths.
 10. The non-transitory computer readablemedium of claim 8, further comprising: determining, by the computingsystem, that the target website underwent an update; inputting, by thecomputing system, further code associated with the updated targetwebsite into the recurrent neural network; and identifying, by therecurrent neural network, a second plurality of additional paths throughthe target web site, the second plurality of additional paths distinctfrom the first plurality of paths prior to the update.
 11. Thenon-transitory computer readable medium of claim 8, further comprising:receiving, by the computing system from the client device, a uniformresource locator (URL) corresponding to target website; inputting, bythe computing system, further code associated with the URL into therecurrent neural network; and identifying, by the recurrent neuralnetwork, a second plurality of paths through the target website, basedon the learned first plurality of paths by skipping one or more sets ofoptions of the target website that correspond to the learned subset ofbroken paths that result in an error.
 12. The non-transitory computerreadable medium of claim 8, wherein the hyperparameters include a speedof switching between states on the website.
 13. The non-transitorycomputer readable medium of claim 8, further comprising: generating, bythe computing system, a plurality of integration tests for at least thewebsite based on the learning.
 14. The non-transitory computer readablemedium of claim 8, further comprising: training, by the computingsystem, the recurrent neural network to learn paths through websitesusing training data sets comprising code associated with the websites.15. A system, comprising: a processor; and a memory having programminginstructions stored thereon, which, when executed by the processor,causes the system to perform operations, comprising: identifying atarget website hosted by a third party server; inputting code associatedwith the target website into a recurrent neural network, wherein agenetic algorithm tunes hyperparameters of the recurrent neural networkthat define how the recurrent neural network simulates a user navigatingthe target website; learning a first plurality of paths through thetarget website in accordance with the genetic algorithm; and based onthe learning, identifying a subset of broken paths that result in anerror.
 16. The system of claim 15, wherein the operations furthercomprise: identifying a uniform resource locator (URL) corresponding tothe target web site; inputting further code associated with the URL intothe recurrent neural network; and identifying, by the recurrent neuralnetwork, a second plurality of paths through the target website, basedon the learned first plurality of paths.
 17. The system of claim 15,wherein the operations further comprise: determining that the targetwebsite underwent an update; inputting further code associated with theupdated target website into the recurrent neural network; andidentifying, by the recurrent neural network, a second plurality ofpaths through the target website, the second plurality of paths distinctfrom the first plurality of paths prior to the update.
 18. The system ofclaim 15, wherein the operations further comprise: identifying a uniformresource locator (URL) corresponding to the target web site; inputtingfurther code associated with the URL into the recurrent neural network;and identifying, by the recurrent neural network, a second plurality ofpaths through the target website, based on the learned first pluralityof paths by skipping one or more sets of options of the target websitethat correspond to the learned subset of broken paths that result in anerror.
 19. The system of claim 15, wherein the operations furthercomprise: generating a plurality of integration tests for at least thewebsite based on the learning.
 20. The system of claim 15, wherein theoperations further comprise: training the recurrent neural network tolearn paths through websites using training data sets comprising codeassociated with the websites.